This invention relates to an information processing system which operators can use to directly handle images displayed on a display unit inserted into a desk and more particularly to an information processing system suited for instinctly handling images using the actual objects placed on the desk. The terms "actual object" and "physical object" have the same meaning in this application unless otherwise indicated.
As written in "(1) Progress in Human Interface Vol. 2, PP. 6572 (1993)", a data glove used to interact with a computer is available as the conventional technique for improving the computer man-machine interface. Also, in the conventional techniques written in (2) Proceedings of UIST'91, "The Digital Desk Calculator: Tangible Manipulation on a Desk Top Display", the hand of an operator without a glove is detected by a camera, the movement of hand and sheets of paper are monitored, display objects are moved, and "cut & copy" is performed by selecting a character string on a sheet of paper with a finger. In addition, in the conventional technique written in (3) "5-19958 Gazette", the hand of an operator is photographed by a camera and used for pointing.
As a prior art of a method for making information handled in an information processing unit corresponding to an object, there is a system named XAX which is indicated in (4) "Bridging the Paper and Electronic Worlds: The Paper User Interface" (Proceedings of INTERCHI 1993, pp. 507 to 512). The object used in this system is a paper on which a discrimination pattern is printed. A user inputs the content written on such a paper into the system using a scanner or FAX. The system recognizes the pattern printed on the paper and discriminates what kind of paper it is. As a discrimination result, the system understands that the paper is a paper on which the content of a file held by the system is printed or the paper is a written order for instructing file retrieval to the system.
When the XAX system can discriminate the paper as a paper on which a file is printed, the system can indicate the original file to the user. From a viewpoint of the user, the printed paper corresponds to the original file and it can be understood that by indicating the printed paper, the original file is taken out. When the read paper is a written order for instructing file retrieval to the system, the system recognizes the information (for example, the file name and keyword) written on the paper and retrieves the corresponding file. The retrieval result is indicated to the user in a certain form. For example, a retrieval result list or the file content can be printed via FAX or a printer. From a viewpoint of the user, the paper (written order) to be used for retrieval corresponds to the retrieval result and it can be understood that by indicating the paper, the retrieval result is taken out.
In the conventional technique which uses the data glove, the man-machine interface is problematic because the operator feels incompatibility. The operator feels incompatibility because the hand of the operator is covered with the glove. Also, in the other two conventional techniques in which the movement of the hand is monitored by the camera and this is used for the man-machine interface, actual operations cannot be performed instinctly because the correspondence between operations of images and actual objects and the functions of instructions for various operations are not considered concretely.
The prior art mentioned above can retrieve information using a paper or give an instruction to an information processing unit but cannot change dynamically the correspondence between the paper and electronic information and limit the operation to the electronic information because it corresponds to the paper. For example, to increase information which can be taken out by a certain paper, it is necessary to additionally write the attribute (for example, the document name) of new information on the paper. Therefore, by this prior art, it is impossible to perform the aforementioned corresponding operation interactively by viewing the screen.
The prior art mentioned above indicates a concept of erasing information which is displayed on the screen using an object but does not give consideration to map and hold the object to each electronic information. Therefore, it is impossible to take out electronic information using an object or to change the attribute of electronic information via the operation to the object.
The present invention is directed toward providing an improved information processing system which establishes an environment in which the operator can interface with computers in a format similar to actual operation and actual objects can be used as man-machine interface parts.
According to one aspect of the present invention, improved processing is achieved by providing for an operator managed menu driven system for storing the physical object attributes of selected actual physical objects to be used. This feature allows for different operator hands, as well as different physical objects. Since actual physical objects to be used are also used in setting up and storing characteristic features to be compared during information processing, an especially user friendly man-machine interface environment is achieved.
According to preferred embodiments of the present invention, there are provided a method for recognizing the actual object from an image of a camera device; an attribute definition method for associating operations between the recognized actual object and the displayed image; and an attribute dealing method for comparing a characteristic point coordinate of the recognized actual object with the characteristic point coordinate of another actual object or that of the image displayed on the display surface and executing operations defined by the attribute definition method for the associated image when the comparison result is within a given value. This object is achieved in the display unit, the camera device that photographs the image displayed on the display unit and the actual object placed on the display unit or near it, and the information processor with the processor connected to the display unit and camera device.
Advantages of the invention are also achieved in part by recognizing a user gesture for holding or releasing the image or actual object displayed on the above-mentioned display surface in a gesture recognition method. This function is achieved in the information processor which provides the method for recognizing the hand gesture of the operator from the image input from the camera device and operates the image on the display surface according to the result of this gesture recognition method.
Assume that a facsimile is placed on a desk as an actual object and a data file is opened on the display unit as an image. When the operator moves this data file in the facsimile direction to be placed within a given distance, the information processor performs facsimile send processing on the assumption that the data file is sent in the facsimile. This enables not only an actual object to be used in the man-machine interface but also an instinctive operating environment to be established. In addition, the operator can directly handle an object in a sense of holding by recognizing the image or actual object hold and release gestures as the hand gesture of the operator without specifying a menu and an icon. The man-machine interface is thus remarkably improved.
According to certain preferred embodiments of the present invention, a system is provided for implementing the interactive operator use of objects to retrieve, read, change and file electronically stored information. The system includes a display device, a file storage means for storing electronic information, a discrimination means for discriminating an object indicated by a user, and a link storage means for storing link information which designates the correspondence between the aforementioned object and the electronic information stored in the aforementioned file storage means. A file operation means is provided for reading the electronic information corresponding to the discrimination result of the object by the discrimination means from the file storage means on the basis of the link information and for displaying it on the display device. A link definition means is provided for generating link information or displaying the electronic information corresponding to the discrimination result of the object on the display device, displaying the discrimination result of the object by the discrimination means and the electronic information stored in the file storage means, and designating the correspondence between the selected electronic information among the displayed electronic information and the discrimination result. An attribute discrimination means is provided for displaying the electronic information corresponding to the discrimination result of the object on the display device and discriminating the attribute information added to the object, and a file operation means is provided for executing the Operation on the basis of the discriminated attribute information for the electronic information having the correspondence with the discriminated object.
According to a first characteristic of embodiments of the present invention with a link storage means, electronic information corresponding to the discrimination result of an object by the object discrimination means is read from the file storage means by the file storage means on the basis of the link information stored in the link storage means a displayed on the display device, so that a user can access the electronic information intuitively using the correspondence between the object and electronic information.
According to a second characteristic of the embodiments of the present invention with a link storage means, the discrimination result of an object by the object discrimination means and the electronic information stored in the file storage means are displayed by the link definition means and link information for designating the correspondence between the selected electronic information among the displayed electronic information and the discrimination result of the object is generated, so that a user can generate link information interactively on the screen.
According to a third characteristic of embodiments of the present invention with a link storage means, the attribute information added to an object is discriminated by the attribute discrimination means and the operation on the basis of the discriminated attribute information is executed for the electronic information having the correspondence with the discriminated object by the file operation means. Namely, it is controlled so that unsuitable processing cannot be executed for the electronic information corresponding to the object according to the attribute information including the information for designating whether the electronic information corresponding to the object is allowed to be modified or not, so that it can be avoided that information is erased due to an incorrect operation by a user.
Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of the invention when considered in conjunction with the accompanying drawings.